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PREFACE 


This  Is  one  of  several  RAND  sCudLes  exploring  ways  Co  use  computers 
for  data  analysis  In  weapon  system  management.  This  particular  Investi¬ 
gation  Is  a  product  of  ^ne  fourth  project  of  our  Logistics  Simulation 
Laboratory  (LP-IV),  and  shows  how  a  weapon  modification  may  be  field 
tested  early  In  the  conversion  to  determine  how  It  Is  affecting  main¬ 
tenance  and  operational  programs. 

The  Memorandum  should  Interest  people  In  Che  Maintenance  Engineering 
Directorates  at  both  Hq  USAF  and  Command  levels  who  want  to  measure  the 
cost  and  effectiveness  of  a  modification  program.  Others  who  may  find 
this  research  useful  are  those  studying  new  methods  for  analyzing  weapon 
system  data. 

For  another  study  of  the  same  general  problem,  Interested  readers 
may  wish  to  consult  K,  L.  Deavers,  Product  Improvement  Candidate  Selection 
and  Management  Information.  The  RAND  Corporation,  RM-4324-PR,  November 
1964. 
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SUMMARY 

For  a  niunber  of  years,  The  RAND  CorporacLon  has  been  studying 
ways  to  increase  the  productivity  of  the  Air  Force  maintenance  man¬ 
agement  system.  These  studies  have  suggested  adding  clock-hours  and 
status  data  to  the  current  data  system*  (AFH  66-1,  Maintenance  Man¬ 
agement)  ,  and  have  proposed  the  use  of  computers  In  maintenance 
management  and  analysis.**  These  additions  have  made  practicable 
analyses  of  maintenance  data  that  were  heretofore  deemed  Impossible. 

This  Memorandum  continues  the  previous  line  of  Inquiry.  It  demonstrates 
how  these  adjuncts  to  the  present  system  may  be  used  to  assess  the 
cost  and  effectiveness  of  engineering  modifications. 

We  first  describe  the  procedures  using  two  examples:  an  engine 
modification  of  T-38A  aircraft,  and  a  fire-control  system  modification 
of  F-IOIB  aircraft.  The  two  procedures  are  then  contrasted  to  show 
how  the  combination  of  methods  can  provide  the  most  meaningful  set  of 
data. 

Next  we  show  how  the  met'aod  of  recording  and  analyzing  may  also 
be  used  to  provide  a  history  of  the  Installation-checkout  process 
during  this  vital  period.  It  Is  worth  comment  that  Information  of 
this  kind  —  elapsed  time  for  completion,  job  standards,  test-flights 
required,  kinds  of  defects  discovered,  bit  and  piece  consumption  and 
man-hour  utilization  --  Is  the  most  difficult  to  obtain  during  the 
confusion  of  a  heavy  modification  program. 

Finally,  we  briefly  discuss  some  factors  that  would  be  Involved 
to  complete  the  package  described.  The  major  addition  to  the  present 
method  would  be  the  Inclusion  of  mission  data  In  z  manner  that  could 
be  related  to  the  maintenance  data. 


★ 

C.  F.  Bell,  Jr.,  and  T.  C.  Smith,  The  Oxnard  Base  Maintenance 
Management  Improvement  Program .  The  RAND  Corporation,  RM-3370-PR, 
November  1962. 
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I .  INTRODUCTION 


Because  of  changing  threat  assessment  and  because  of  wear -out 
factors,  a  weapon  cannot  be  tested  under  all  circumstances  before  it 
comes  into  the  operational  inventory.  As  experience  with  the  weapon 
reveals  the  need  for  improvements,  they  are  developed,  contracted  for, 
and  scheduled  into  the  system.  This  is  the  weapon  modification  program. 

Modifications  are  of  three  general  types:  (1)  the  basic  repair, 
concerned  with  the  prevention  of  wear-out;  (2)  the  reliability-main¬ 
tainability  improvement  modification,  most  often  concerned  with  high- 
failure  components;  and  (3)  the  performance  improvement  modification, 
concerned  with  improving  mission  capability  or  improving  safety-of- 
flight.  This  Memorandum  deals  mainly  with  the  latter  two  types. 

It  is  difficult  to  estimate  exactly  what  a  modification  program 
costs,  but  it  can  be  considerable.  For  example,  in  1962,  Oxnard  Air 
Force  Base  averaged  728  aircraft  hours  per  month  in  Time  Compliance 
Technical  Orders  (TCTO's)  as  reported  by  AFR  65-llOA  procedures. 

This  is  the  equivalent  of  one  aircraft  in  continuous  TCTO  status, 
which  represents  a  loss  in  operational  capability.  Also,  during  the 
modification  program  described  later,  Oxnard  was  excused  from  active 
air  defense  for  two  months.  This  represents  a  severe  loss  in  opera¬ 
tional  capability. 

In  terms  of  man-hour  costs,  the  amounts  can  become  substantial. 

For  example,  at  one  time.  Air  Defense  Command  (ADC)  requested  an 
average  of  SO  man-hours  a  month  per  possessed  aircraft  for  modification 
purposes.  Hence  a  squadron  of  20  aircraft  would  average  2000  man-hours 
each  month . 

That  the  modification  program  is  costly  is  apparent.  The  way  cost 
is  defined,  of  course,  is  most  germane.  Although  there  are  many  ways 
to  define  cost,  the  definition  used  here  includes  three  factors: 

1.  The  dollar  cost  of  kits  and  engineering  (Including  technical 
manuals  . 

2.  The  man-hour  costs  for  installation  and  checkout,  with  civilian 
and  military  personnel  kept  separate. 
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3.  The  mission  Inventory  loss,  i.e.,  some  representation  of 

the  reduction  of  mission  effectiveness  due  to  non-availability 
of  the  weapon  during  modification.  This  is  generally 
expressed  in  two  ways 

a.  as  a  whole  number,  i.e.,  aircraft  hours,  and 

b.  as  a  ratio  of  this  number  to  weapon  inventory, 
i.e.,  aircraft  hours  lost  to  tech-order  programs 
divided  by  possessed  aircraft  hours. 

To  set  our  general  dimensions  right,  let  us  briefly  note  the  costs 
of  two  recent  engineering  modification  programs.  In  1963,  Air  Training 
Command  modified  the  power  plants  of  all  T-38A  aircraft.  The  cost  of 
installation  kits  and  engineering  was  $1.7  million.  The  average  military 
man-hours  used  for  installation  and  checkout  was  275  to  300  per  aircraft. 
In  addition,  a  private  firm  was  paid  $15  million  to  assist  in  the  instal¬ 
lation.  No  data  were  kept  on  the  loss  of  the  weapon  from  the  operational 
inventory. 

During  1963,  Air  Defense  Command  did  a  fire  control  modification  of 
F-lOl's.  The  engineering  and  kit  cost  was  $21.9  million.  No  man-hour 
figures  were  available  on  the  installation,  which  was  done  at  the  depot. 
The  alignment  and  checkout  was  done  in  the  field,  and  averaged  450 
man-hours  per  aircraft  (of  this,  two-thirds  was  flight-line  maintenance 
and  one-third  was  bench  repair).  The  aircraft  Inventory  loss  was  approx¬ 
imately  five  weeks  per  aircraft:  three  weeks  for  installation;  one  week 
for  check-out  and  alignment;  and  one  week  for  problems  of  scheduling, 

N0RS,  and  transportation. 

Of  course,  not  all  modifications  are  of  such  magnitude.  The  majority 
are  much  more  pedestrian  in  nature:  replacing  diodes  to  get  better  peak 
inverse  voltage  tolerances,  rewiring  to  reduce  high-voltage  arcing, 
fusing  circuits,  increasing  ventilation,  etc.  Although  these  are  far 
less  dramatic,  each  involves  engineering  and  installation  costs,  and 
each  installation  generally  results  in  some  loss  to  the  mission  inven¬ 
tory.  The  accumulation  of  these  small  items  typically  adds  up  to  a 
formidable  total. 

In  costing  a  modification  program,  then,  there  are  at  least  three 
areas  that  should  be  considered:  the  part's  dollar  cost,  the  man-hour 
cost,  and  the  loss-of-mission  cost.  The  assessment  of  costs,  however, 
is  only  part  of  the  task.  A  modification  is  Installed  to  obtain  one. 


r 
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or  more,  specific  objectives.  The  critical  question  becomes:  To  what 
extent  did  the  modification  accomplish  its  stated  objective?  This 
brings  up  the  issue  of  how  to  measure  effectiveness,  the  concern  of  a 
major  part  of  this  Memorandum. 

Since  most  modifications  represent  an  effort  to  improve  either 
equipment  reliability  or  mainti^’.nability ,  the  major  part  of  the  effort, 
to  date,  has  been  devoted  to  obtaining  measures  in  the  reliability- 
maintainability  area.  More  specifically,  the  major  effort  has  been 
directed  toward  getting  break-  and  recovery-rate  distributions,  i.e., 
both  the  means  and  the  variances.  Knowledge  of  the  variances  is  nec¬ 
essary  if  we  are  to  make  definite  statements  about  the  impact  of  the 
modification  on  effectiveness.  This  point  will  become  apparent  in  the 
following  discussion. 
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H.  THE  SORTIE  AS  THE  UNIT  MEASURE 

Early  In  Che  development  of  a  package  of  computer  programs  to  be 
used  In  analyzing  maintenance  data,  it  became  evident  that  Che  ability 
to  isolate  data  by  the  specific  sortie  was  the  sine  qua  non  of  any 
analysis  in  depth.*  A  specific  sortie  is  one  flown  on  a  specific  day 
and  hour  by  a  specific  tail  number.  The  data  we  wanted  to  isolate  by 
sortie  involved  determining:  the  kind  of  mission  flown,  the  systems 
and  subsystems  that  required  unscheduled  maintenance,  and  the  time  and 
man-hours  consumed  in  maintenance. 

Specific  sortie  analysis  contrasts  with  general  sortie  analysis 
in  Chat  the  latter  deals  with  aggregates  (generally  monthly  totals). 
There  is  a  tremendous  difference  in  the  amount  of  information  each 
method  yields.  Take  a  typical  example  --  the  unscheduled  man-hours 
per  sortie  cost.  In  Che  case  of  the  generalized  sortie,  Che  total 
unscheduled  man-hour  cost  is  determined  and  then  divided  by  Che  number 
of  sorties  flown.  The  result  is  the  average  number  of  man-hours  per 
sortie,  a  single  quantity. 

In  analyzing  the  specific  sortie,  Che  unscheduled  man-hour  cost 
is  determined  by  processing  the  individual  sortie  measures.  Hence, 
it  is  possible  to  get  not  only  the  mean  man-hours  per  sortie,  but  also 
the  variation  around  this  mean.  When  this  is  done,  it  quickly  becomes 
apparent  that  not  all  sorties  require  unscheduled  maintenance.  Hence 
two  sets  of  data  can  be  obtained:  those  that  include  the  "unscheduled 
maintenance  required"  information,  and  Chose  that  do  not.  As  will  be 
shown,  both  sets  of  measures  are  important,  particularly  when  dealing 
with  mission-related  questions. 

It  is  also  apparent  that  a  tabulation  of  "unscheduled  maintenance 
required"  sorties  provides  the  data  for  computing  break-rates,  which 
are  the  proportion  of  sorties  requiring  unscheduled  maintenance.  These 
two  sets  of  data,  the  frequency  of  breaks  and  Che  duration  of  recovery, 
provide  tremendous  leverage  on  problems  of  scheduling  and  operations. 
Further,  since  Che  specific  sortie  data  enable  the  computation  of  Che 
variances,  Che  methods  of  statistical  analysis  may  be  used,  considerably 
enhancing  Che  analytic  process. 

A.  Sweetland,  The  Use  of  Computers  in  Air  Force  Maintenance 
Management  and  Analysis.  The  RAND  Corporation,  RM-4228-PR,  October  1964. 
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Once  we  realized  how  important  specific  sortie  data  are,  we  began 
developing  a  computer  program  (completed  in  early  1963)  to  summarize  the 
act  of  recovery.  The  program  accumulates,  for  each  sortie:  the  man¬ 
hours,  the  elapsed  time  (for  the  recovery),  the  units  produced  and 
break-rate  for  each  set  of  unscheduled  maintenance  for  the  recovery  of 
the  entire  aircraft,  for  eac i  system  (two-digit  work  unit  code),  sub¬ 
system  (three-digit  work  uni?  code)  ,  and  each  work  center.* 

The  data  describing  the  kind  of  sortie  are  also  included  in  each 
record.  Thus  we  have  not  only  the  break-rate  and  recovery  data  men¬ 
tioned  previously,  but  also  the  associated  mission  data.  Hence,  we 
have  most  of  the  information  necessary  to  investigate  the  effects  of 
a  modification,  i.e.,  we  can  determine  the  effect  on  the  mission,  the 
break-  and  recovery-rates,  and  the  cost  (in  units  and  man-hours). 

The  missing  element  in  the  measurement  is  the  determination  of 
how  effective  the  modification  is  (unless  the  objective  is  to  Improve 
the  mission,  break-rate,  recovery  or  cost  figures).  For  example,  many 
modifications  are  installed  for  the  comfort  and  efficiency  of  the  crew. 
In  these  instances,  it  is  not  only  desirable  but  mandatory  to  construct, 
a  priori .  a  set  of  measures  to  determine  the  impact  of  the  modification. 

As  will  be  seen,  the  missing  element  in  the  following  examples  is 
a  set  of  measures  to  determine  how  effective  the  modification  is. 

Since  our  purpose  is  to  demonstrate  the  feasibility  of  a  methodology, 
this  omission  is  not  a  serious  defect.  But  for  an  actual  cost- 
effectiveness  analysis,  the  omission  would  be  intolerable. 

The  procedures  of  the  mod-costing  process  divide  logically  into 
three  parts: 

1.  The  pre-planning  phase  that  includes  preparations  for  the 
mod-installation.  Cardinal  among  these  is  the  design  of 
a  measure  to  determine  the  effect  of  the  modification. 

2.  The  installation  phase,  consisting  of 

a.  the  installation 

b.  the  checkout-alignment. 

3.  The  post-installation  phase  that  determines  the  comparative 
effects  of  the  modification. 


Ibid .  pp  .  A  . 
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We  will  discuss  these  three  elements  beginning  with  item  three, 
end  working  backwards  --  since  this  will  make  the  process  considerably 
easier  to  comprehend*  The  two  real-life  exercises  already  mentioned 
will  be  used  in  illustration:  a  power  plant  modification  (system  23XXX) 
done  on  T-38A  aircraft  at  Williams  Air  Force  Base  (ATC) ,  and  a  fire- 
control  modification  (system  74AXX)  done  on  F-IOIB  aircraft  at  Oxnard 
Air  Force  Base  (ADC) . 


III.  TWO  EXAMPLES  OF  MODIFICATION  FIELD  TESTING 


In  our  first  attempt  to  develop  a  method  of  costing  the  effects 
of  a  modification,  we  used  a  sample  of  Williams  Air  Force  Base  data. 

Since  the  venture  was  on  an  a  posteriori  basis,  there  was,  necessarily, 
a  good  bit  of  make-do  involved.  We  wanted  to  know  what  impact  a  fairly 
expensive  and  extensive  power-plant  modification  had  on  the  T-38A  air¬ 
craft.  The  sample  was  taken  from  data  for  the  last  two  weeks  of  July 
1963,  when  the  possessed  aircraft  count  was  84,  and  the  sorties  flown 
totalled  1358. 

Inspection  showed  the  sample  data  :ould  be  subdivided  into  three 
groups: 

1.  A  control  group  of  unmodified  aircraft  (CTL) . 

2.  An  experimental  group  modified  during  the  months  of  May  and 
June  (EXF) .  This  group  would  not  be  expected  to  have  all  the 
bugs  eliminated  since  the  average  age  of  installation  was 
approximately  six  weeks. 

3.  An  experimental  group  modified  for  sufficient  time  to  have 
stabilized  (EXM)  •  These  were  modified  between  December  1962 
and  April  1963.  The  average  age  of  installation  was  about 
three  and  one-half  months. 

The  data  were  divided  into  these  groups  and  then  processed  by  the  re¬ 
covery  program  to  yield  the  sortie  summary  data.  These  latter  were 
analyzed  in  a  number  of  ways,  as  described  and  shown. 

The  first  thing  of  interest  is  the  break-rate,  which  identifies 
(for  both  system  and  subsystem)  the  number  of  sorties  that  were  follow¬ 
ed  by  unscheduled  maintenance.  The  break-rates  for  the  three  aircraft 
groups  are  compared  in  Fig.  1.  In  this  figure,  the  statistical  analy¬ 
sis  indicates  that,  although  the  "trend"  appears  to  drop  from  10/100  to 
to  8/100,  the  trend  may  be  attributed  to  random  variation.  Even  when 
the  test  is  computed  via  the  more  sensitive  arc  sine  method,  there  is 
still  no  justification  for  assuming  that  the  variation  among  the  measures 
is  other  than  random. 

Following  the  break-rate  analysis,  three  separate  avenues  were 
explored:  (1)  the  analysis  of  recovery  times  (system);  (2)  the  analysis 
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of  man-hour  data  (system);  and  (3)  the  analysis  of  the  recovery  of 
the  entire  aircraft.  Only  the  first  of  these,  individual  system 
recovery,  is  discussed  in  detail  here.  Examples  of  points  two  and 
three  are  contained  in  the  Appendix. 

The  recovery  data  shown  in  Fig.  2  were  tested  in  two  ways:  with 
and  without  the  inclusion  of  the  "no-break"  data.  Again,  we  find  a 
trend  in  the  desired  direction.  But  again  we  find,  as  with  the  break- 
rate  data,  that  random  variation  may  account  for  the  changes.  The 
data  in  Fig.  2,  incidentally,  provide  one  form  of  job  standard  to  be 
used  in  managing  the  flight-line  (also  see  Fig.  8). 

By  adding  the  no-break  data  to  the  data  shown  in  Fig.  2  we  deter¬ 
mine  the  impact  on  the  "average"  sortie.  Combining  the  no-break  and 
break  data,  in  effect,  combines  the  information  of  Figs.  1  and  2. 

These  average  sortie  data  are  important  from  the  viewpoints  of  both 
planning  and  operations.  In  terms  of  capability  forecasts,  predicted 
recovery,  projected  kills,  etc.,  these  data  show  what  can  be  expected 
"on  the  average."  For  example,  if  we  are  trying  to  squeeze  training 
sorties  out  of  our  possessed  aircraft,  these  numbers  give  us  planning 
figures  --  expected  losses  to  unscheduled  maintenance. 

The  data  of  Fig.  1  show  a  decreasing  trend  in  frequency  of  un¬ 
scheduled  maintenance,  while  the  data  of  Fig.  2  show  a  decreasing  trend 
in  the  time  needed  to  recover  the  broken  system.  The  interesting  ques¬ 
tion  is  whether  the  two  statistically  insignificant  trends  may  co.ubine; 
Fig.  3  shows  the  results  of  such  efforts.  Combining  the  trends  has  no 
effecc  on  the  conclusion  that  only  random  variation  is  operating.  The 
differences  among  the  means  have  no  significance. 

The  findings  on  the  T-38A  break-rate  and  recovery  data  were  cor¬ 
roborated  with  the  man-hour  data  (Appendix,  Figs.  4A  and  5A) .  We  also 
cross-checked  the  three-digit  data  (subsystem)  with  the  recovery  data 
of  the  entire  aircraft  (Appendix,  Figs.  lA  to  3A) .  The  results  were 
essentially  the  same  --  a  trend  in  the  indicated  direction  too  small 
to  have  any  statistical  significance. 

But  had  there  been  statistical  significance,  there  would  still 
have  been  no  practical  significance.  For  example,  the  "savings" 
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Fig.  3  --  T-38A  Fire-Control  Recovery  Tlae 
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resultlng  from  the  modification  amounts  to  0.15  man-hours  a  sortie. 

Using  an  average  man-hour  cost  of  $3.54,*  the  dollar  savings  amounts 
to  approximately  83  cents  a  sortie.  Even  with  the  fantastic  number 
of  sorties  that  Williams  Air  Force  Base  flies  (1358  in  two  weeks),  the 
cost  of  the  modification  probably  would  not  be  amortized  within  the 
life  of  the  system.  (With  the  current  flying  schedule,  the  savings  on 
each  aircraft  are  $200  to  $300  a  year.  The  cost  to  install  was  275  to 
300  man-hours  per  aircraft,  i.e.,  about  $1660.) 

We  should  establish  one  point  firmly:  we  have  been  speaking  of 
the  impact  on  the  recovery.  For  the  T-38A,  recovery  was  not  the  pri¬ 
mary  objective  of  the  modification.  The  primary  objective  was  to 
eliminate  ground  engine  stalls,  promoting  ease  of  operation.  The 
secondary  objective  --  to  improve  the  recovery  picture  a  d  thus  increase 
the  number  of  training  sorties  --  was,  in  effect,  an  afterthought. 

The  T-38A  data  has  the  desirable  feature  of  permitting  comparison 
of  experimental  and  control  groups  sampled  at  the  same  point  in  Lime 
(i.e.,  the  last  two  weeks  in  July).  The  shortcoming  of  the  data  is 
that  they  do  not  provide  a  means  of  observing  the  changes  over  time, 
i.e.,  under  a  variety  of  stresses.  The  Oxnard  fire-control  modification 
provided  the  reverse  --  absence  of  a  control  group  from  the  same  point 
in  time,  but  providing  a  comparison  across  time.  The  advantages  and 
disadvantages  are  also  reversed.  The  contrast  between  the  two  sets  of 
data  will  highlight  a  point  worth  making  --  that  the  really  desirable 
method  involves  contrasting  experimental  and  control  groups  across  time. 

The  Oxnard  F-IOIB  fire-control  modification  was  designed  to  im¬ 
prove  low-altitude  interception  capability  (primary  objective),  with 
no  more  than  a  5-per-cent  Increase  in  break-rate  and  recovery  times 
(secondary  objective).  As  with  the  previous  example,  we  shall  consider 
only  the  secondary  objective. 

The  January  through  May  period  was  a  time  of  manual  operation  at 
Oxnard  Air  Force  Base,  and  thus  serves  as  a  base-line.  Beginning  June 
17th  the  aircraft  were  sent,  serially,  to  the  depot  for  Installation  of 
the  modification.  On  completing  the  installation,  the  system  was  given 


AFM  400-12,  28  February  1964,  Paragraph  79  F(3).  This  factor 
expresses  direct  man-hours  and  does  not  include  overhead. 
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a  "smoke  test"  ("...  turn  it  on  and  if  you  do  not  see  or  smell  anything, 
send  it  back")  and  returned  to  Oxnard  for  checkout  and  alignment.  The 
last  aircraft  was  released  (returned  to  Operations)  August  13th.  By 
October,  the  system  had  stabilized  from  the  effects  of  the  program. 

(Although  the  June-September  data  are  included,  only  the  data  from 
October  on  should  be  compared  with  the  January-May  base-line  when  mak¬ 
ing  the  critical  determination  of  the  impact.) 

Figures  4,  5  and  6  show  the  same  information  as  Figs.  1,  2,  and  3  -- 
the  impact  on  break-rates  and  recovery  times.  The  data  indicate  an  in¬ 
crease  in  break  and  recovery  (over  the  base  of  January-May)  followed 
by  a  return  to  within-bounds  condition  in  January  of  1964.  These  re¬ 
sults  were,  of  course,  cross-checked  with  the  man-hour  and  aircraft 
recovery  data  (not  included).  All  analyses  yield  essentially  the  same 
information. 

To  summarize  the  F-101  analysis,  we  may  tentatively  say  that  the 
data  suggest  the  modification  has  had  little  impact  on  break-  and 
recovery-rates.  But  we  cannot  make  a  firm  statement  since  we  are  at  a 
disadvantage  in  trying  to  compare  data  from  different  points  in  time. 

It  is  said  that  no  month  (at  an  air  base)  is  typical,  and  herein  lies 
the  problem.  Oxnard,  for  example,  underwent  a  complete  change  in 
command  philosophy  in  the  sample  period,  going  from  a  centralized  to 
a  decentralized  control.  This  might  be  the  the  cause  of  our  differences. 

Then  too,  there  is  a  tremendous  month- to-month  variation  in  the 
number  of  sorties  flown,  and  we  have  observed  a  negative  correlation 
between  sorties  flown  (per  available  aircraft)  and  break-rates,  as  well 
as  sorties  flown  and  recovery  time.  Further,  there  are  variables  such 
as  weather,  TOC  requirements,  training  requirements,  special  exercises, 
fly-bys,  etc.  that  disrupt  any  even  distribution  of  schedules.  Hence, 
serious  doubts  arise  when  comparing  data  across  time  because  of  the 
problem  of  differing  forces  that  may  be  affecting  them. 

We  insist  here  that  the  only  way  to  obtain  any  meaningful  infor¬ 
mation  is  to  have  data  samples  (experimental  and  control)  taken  simul¬ 
taneously,  as  with  T-38A  data,  under  a  wide  variety  of  operating  conditions, 
as  with  the  F-lOl  data.  If  we  cannot  have  both,  we  would  prefer  to 
have  the  simultaneous  sample.  The  reasons  for  this  last  statement 
become  evident  by  assuming  that  the  F-lOl  numbers  had  shown  true  and 
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•  Ignlf leant  differences:  our  dileni&a  then  would  be  to  determine 
whether  the  differences  were  due  to  the  modification  per  se,  the 
varying  forces,  or  a  combination  of  the  two. 
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(Ttits  figure  !■  slmtler  to  Fig.  1  but  Includes  more  Infonnation.  The  base-line  gives  the  sorties- 
flovn  cMnt,  TO  is  the  observed  frequency  of  breaks.  The  theoretical  frequencies  (FT),  and  the 
(FO-n)^/PT  entries  are  part  of  the  conventional  chi-square  computations.  They  are  included  to 
facilitate  analysis.  The  means  column  (FO/base-llne)  has  asterisk  tags  beside  those  Itms  that 
show  significant  differences  between  the  FO  and  FT  columns.) 
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(January  1963  —  SMpla  I,  Hanth  01;  through  January  1964  —  Sanple  12,  Month  01.  The 
P-Teat  la  highly  algnlflcant.  Hote  that  the  October  naan  la  well  within  the  January- 
May  range.  The  no-break  data  are  not  Included.  Aa  In  Pig.  2,  frequency  count*  are 
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IV.  MONITORING  INSTALLATION  AND  CHECKOUT 


To  facilitate  the  fire-control  checkout  at  other  bases,  the  28th 
Air  Division  requested  that  Oxnard  provide  modification  data  based  on 
the  first  12  aircraft  it  released  back  to  Operations.  The  list  of 
Oxnard  contributions  included  among  other  things: 

1.  The  "queue  sorts"  on  each  aircraft  --  a  pictorial  history 
of  the  alignment  and  checkout  procedures 

2.  The  elapsed  time  for  completing  the  checkout  process  for 
each  aircraft 

3.  The  man-hours  and  units  produced  for  each  aircraft 

4.  The  total  bit  and  piece  consumption  for  all  12  aircraft 

5.  The  number  of  test-flights  necessary  to  learn  whether  the 
new  equipment  was  functioning  satisfactorily 

6.  The  kinds  of  defects  discovered  in  the  new  system  (i.e., 
flight-crew  write-ups) 

7.  The  revised  job  standards. 

Many  of  these  items  were  satisfied  by  a  rather  peculiar  kind  of 
listing.  In  the  beginning,  we  created  an  output  that  described  the 
checkout  process  almost  entirely  in  English.  One  of  the  computer  pro¬ 
grams  we  had  developed  provided  for  the  inclusion  of  in-English  state¬ 
ments  that  were  key-punched  and  tagged  with  a  unique  card- code.  In 
producing  the  daily  monitoring  program,  the  computer,  on  sensing  this 
code,  printed  the  comment  verbatim. 

In  this  new  output,  we  instructed  the  computer  to  print  the  verbal 
statement  along  with  the  accumulated  units  and  man-hour  data  for  each 
aircraft.  The  result  was,  essentially,  a  word-history  of  the  checkout 
process.  A  page  of  this  product  is  shown  in  Fig.  7,  which  gives  the 
highlights  of  the  checkout  for  three  aircraft  --  tail  numbers  7434, 

7436  and  7438. 

The  time  entries  locate  each  event.  The  elapsed  time  of  installa¬ 
tion  is  obtained  by  subtracting  time  begun  from  time  end,  and,  where 
necessary,  adjusting  for  NORS  condition.  The  flight  crew  write-ups 
are  shown  for  each  sortie  ("not  released"  indicating  not  released  back 
to  operations.)  The  units  produced  and  man-hour  totals  conclude  the 
summary  for  each  aircraft. 
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Job  standards  were  computed  for  each  of  the  five-digit  work-unit 
codes  in  subsystem  74AXX.  Only  one  of  these  is  shown,  the  amplifier 
relay  assembly,  work-unit  code  7444E.  Figure  8  depicts  the  Job  stand¬ 
ards  program  output  --  two  juxtaposed  histograms  identical  except  for 
each  of  the  cell  entries.  Each  similarly  located  pair  of  cells  repre¬ 
sents  one  200  series  document.  The  program  produces  an  approximation 
of  job  duration  under  a  wide  variety  of  actions  taken.  Since  these 
data  are  inevitably  of  the-standard-deviation-equals- the-mean  variety 
(i.e.,  no  possibility  of  predicting  their  duration),  the  probability  of 
being  completed  by  various  points  in  time  is  computed  (the  "P-"  line 
under  the  left  histogram.) 

Figure  8  well  illustrates  the  capability  of  the  job  standards 
program  to  winnow  information  from  a  limited  amount  of  data.  In  it, 
we  see  summarized  the  entire  set  of  actions  on  work  unit  7444E. 
(Incidentally,  the  isolated  "5*'  at  6  hours  is  undoubtedly  a  keypunch 
error  since  "5"  is  the  code  for  "deferred.") 

The  reader  has  undoubtedly  noticed  a  lacunae  in  both  samples:  the 
Installation  data  (i.e.,  the  AFTO  212  information).  In  neither  instance 
was  this  information  available.  The  F-101  modification  was  installed 
at  the  depot.  The  T-38A  data  were  not  available  (i.e.,  occurred  before 
the  inclusion  of  clock-hours  in  the  AFM  66-1  data).  No  difficulty  is 
anticipated  in  processing  the  212  data,  since  the  only  statistical 
accumulation  is  a  man-hour  total  for  each  unit  produced  (i.e.,  instal¬ 
lation).  The  one  addition  to  the  regular  66-1  system  is  the  determination 
of  installation  elapsed  times.  These  can  be  obtained  by  any  of  four 
methods:  using  the  verbal  inputs  as  in  the  F-101  data,  manually  check¬ 
ing  the  daily  monitoring  products,  or  making  a  special  pass  through 
the  recovery  program.  A  less  elegant,  but  adequate,  method  is  to  divide 
total  man-hours  by  average  team  size. 


f  I<E-01C1T  SYSTEM  T444E 


Fig.  8  --  Job  Standards  for  Che  Amplified  Automatic  Relay 
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V.  FUTURE  DEVELOPMENTS 

At  first  glance,  it  would  seem  that  the  methods  suggested  above 
could  be  used  in  pretesting  a  modification  to  determine  whether  to 
purchase.  Realistically,  such  a  program  is  not  feasible.  The  time 
it  takes  to  develop  a  modification  may  be  (and  often  is)  measured  in 
years.  This  fact,  coupled  with  the  expense,  precludes  any  "back  to 
the  drawing  board"  modus  operandi.  Instead,  modifications  are  bought 
as  "specs"  i.e.,  the  fire  control  was  bought  as  a  specification  "to 
improve  low-altitude  intercept  capability..."  (capability  is  defined 
in  detail).  The  particular  way  the  manufacturers  will  meet  these 
specifications  is  immaterial  at  this  point. 

Developing  hardware  to  meet  the  specification  is  often  costly: 
Indeed,  development  costs  may  be  several  times  as  great  as  the  final 
hardware  .  ^sts.  Hence,  the  interest  in  determining  modification  im¬ 
pact  is  not  to  accept  or  reject  a  product,  but  rather  to  determine 
whether  the  specification  is  met.  If  not,  it  is  reasonable  to  expect 
a  delay  of  acceptance.  In  general,  delay  does  not  occur  since  speci¬ 
fications  were  bought  rather  than  hardware. 

The  previous  paragraph  is  not  meant  to  imply  that  the  process  is 
uneventful.  Quite  the  contrary.  It  is  inevitably  difficult  and  often 
impossible  to  establish  reliable  and  valid  operational  definitions  on 
an  a  priori  basis.  Even  the  simple  T-38A  specification,  "...the  elim¬ 
ination  of  J-85  engine  stalls,  ground,"  demands  some  thought.  How  long 
must  an  engine  have  run  before  a  stoppage  can  be  called  a  stall,  two 
seconds?  ten  seconds?  ten  minutes?  We  have  all  had  the  experience  of 
having  an  automobile  cough  twice  and  die.  If  an  aircraft  does  this, 
does  it  constitute  a  stall?  If  immediate  restart  is  possible,  is  the 
stall  as  significant  as  a  stall  that  occurred  when  taxiing  (i.e.,  dis¬ 
connected  from  ground  support  equipment).  The  questions  appeared 
trivial  until  we  realized  that  about  $17  million  was  Involved. 

Despite  these  apparent  difficulties,  it  does  seem  possible  to 
define  a  set  of  terms  that  are  reasonably  unequivocal.  And,  once 
these  are  had,  to  record  the  number  of  engine  stalls  under  various 
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conditions:  taxi,  test-cell,  run-up,  take-off,  etc.  Also,  since  the 
overwhelming  majority  of  tech-orders  are  concerned  with  system  or  com¬ 
ponent  reliability,  the  possibility  of  obtaining  by-sortie  data  implies 
that  before-and-af ter  comparisons  are  feasible,  and  further,  that  we 
can  statistically  determine  whether  the  differences  are  real  or  are 
random  artifacts.  The  need  for  adequate  control  groups  (as  in  the 
T-38A  engine  analysis)  is  re-emphasized. 

Measuring  the  improvement  in  direct  mission  effectiveness  is,  of 
course,  impossible,  since  the  only  true  measures  are  those  taken  from 
actual  combat,  and  we  have  no  desire  to  validate  such  a  set  of  measures. 
Instead,  we  shall  have  to  be  content  to  infer  the  results  from  measures 
taken  in  peacetime  conditions.  Thus,  it  seems  reasonable  to  conclude 
that  an  interception  modification  that  reduces  peacetime  break-  and 
recovery-rates  would  also  improve  combat  kill  probabilities.  We  would 
like,  however,  some  more  direct  measures  of  mission  accomplishment: 
how  good  was  the  bomb  drop?  Would  the  air-to-air  missile  have  killed 
the  target?  At  present,  at  least  two  such  measures  show  promise:  the 
SAC  U-82  reports  for  aircraft,  GAMS,  and  missiles  (see  SACK  66-7),  and 
the  ADC  76-3  reports  for  intercept  success  (see  ADCR  66-28).  Both 
documents  are  particularly  admirable  because  they  permit  retrieval  of 
"total- items- at  temp ted- and- total- items- successful"  information.  Hence , 
the  data  are  amenable  to  statistical  methods.  In  addition,  the  ADC 
document  describes  in  detail  the  circumstances  and  characteristics  of 
both  target  and  interceptor  at  the  time  of  intercept  attempt.  Thus, 
one  can  determine  what  kinds  of  tactics  yield  the  best  kill  ratio  for 
a  wide  variety  of  circumstances. 

Both  the  SAC  and  ADC  reports  have  a  serious  deficiency,  however: 
it  is  nearly  impossible  to  relate  the  data  to  the  associated  maintenance. 
Without  special  techniques,  one  cannot  determine,  for  example,  what 
methods  of  maintenance  result  in  degraded  mission  performance,  or  what 
material  failures  are  specifically  responsible  for  the  loss  of  the 
mission.  Note  that  the  latter  is  obtainable  from  the  narrative  accounts, 
but  in  no  instance,  is  this  information  available  to  the  Logistics 
Command . 


Hence,  there  is  need  for  prolonged  exploration  to  relate  the 
mission  and  maintenance  data  and,  thereby,  to  assess  the  impact  of 
material  and  maintenance  on  the  mission,  and  the  inverse  impact  of 
the  mission  on  material  maintenance.  This  exploration  should  result 
in  the  development  of  better  methods  for  determining  modification  im¬ 
pact  and,  more  important,  the  suggestion  of  modifications  that  would 
improve  mission  effectiveness. 

There  is  another  area  that  needs  considerable  exploration:  the 
determination  of  those  factors  affecting  the  break-  and  recovery- rates. 
For  example,  it  is  well-known  among  ADC  maintenance  people  that  high- 
altitude  (over  50,000  feet)  intercepts  result  in  increased  power  plant 
and  radar  problems.  Current  unpublished  studies  show  a  score  of  such 
factors  operating,  the  most  predominant  one  being  the  human  decision 
factor.  There  is  a  need  for  a  complete  exploration  to  determine  what 
factors  are  the  dominant  affecters  of  break-  and  recovery- rates ,  in 
order  to  make  more  meaningful  management  decisions  such  as  "what  re¬ 
sources  to  preposition  under  what  circumstances." 

A  third  area  to  explore,  test  design,  can  be  expected  to  make  a 
substantial  contribution  toward  creating  more  adequate  methods  of  meas¬ 
urement.  Over  the  years,  a  large  number  of  mathematical  and  statis¬ 
tical  methods  have  been  developed  to  enable  the  construction  of  a  variety 
of  tests  (such  as  psychological  testing  for  I.Q. ,  emotional  stability, 
vocational  interest,  and  vocational  aptitude;  or  industrial  testing 
for  process,  control,  acceptance  inspection,  and  reliability.) 

The  more  we  work  with  the  problem  of  maintenance  analysis,  the  more 
we  are  realizing  that  we  have  available  a  large  collection  of  rigorous 
methods  for  helping  us  evaluate  the  effectiveness  of  maintenance, 
material,  and  operations.  Some  of  the  more  recent  developments  in 
multi-variate  techniques  appear  to  be  particularly  promising  because 
they  facilitate  the  evaluation  of  the  effects  of  simultaneous  impact 
of  a  wide  variety  of  elements. 
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VI.  CONCLUSIONS 


Given  some  comparatively  minor  additions  to  current  Air  Force 
documentary  procedures,  major  increases  in  quality  and  depth  of  analysis 
of  maintenance  material  problems  are  possible.  In  this  Memorandum  we 
have  shown  that  the  ability  to  isolate  information  by  the  unique  sortie 
provides  break-  and  recovery-rate  data  not  previously  available.  The 
usefulness  of  these  data  was  demonstrated  by  two  cases  --  a  T-38A  en¬ 
gine  modification  and  an  F-101  radar  modification.  It  was  further 
concluded  that  mission  data  must  be  available  in  a  form  that  can  be 
related  to  maintenance  data  in  order  to  make  a  complete  picture. 


APPENDIX 


This  Appendix  contains  Williams  Air  Force  Base  T-38A  analyses. 
Figures  lA  to  3A  correspond  to  Figs.  1  to  3  In  the  text.  The  data 
show  the  recovery  of  the  entire  aircraft.  Figures  4A  and  5A  show  the 
man-hours  used  to  correct  power-plant  failures. 

Similar  tests  were  also  made  with  the  unlts-produced  data.  It 
should  be  noted  that  there  are  high,  positive  correlations  among  these 
measures  (recovery  times,  man-hours,  and  units  produced).  Hence  they 
cannot  be  viewed  as  Independent  evaluations. 


28 


Fig.  lA  --  T-38A  Br««k-riitca  CoapuCed  for  ch£  Entire  Aircraft 

(All  Syateas  Included) 
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(Shows  mean  time  used  Co  recover  the  entire  aircraft. 


MAFB  T-3a*  AI«CKAFT  RECOWEHr  TIMES  CODE  «  A/C 

FREOUENCV  COUNTS  MCANS  N  TOTALS 
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(Break-data  only.  Because  of  Che  high  correlation  between  recovery 
time  and  man-hours,  these  data  contribute  little  additional  Information. 
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